Geometric conditions which imply compactness of the c^- Neumann 

operator* 

Emil J. Straube 

Department of Mathematics, Texas A&M University, College Station, TX 77843 

straube@matli.tamu.edu 

Abstract 

For smooth bounded pseudoconvex domains in C^, we provide geometric conditions on (the 
points of infinite type in) the boundary which imply compactness of the 9- Neumann operator. It 
is noteworthy that the proof of compactness does not proceed via verifying the known potential 
theoretic sufficient conditions. 



Let be a smooth bounded pseudoconvex domain in C". The (9- Neumann operator N on 
(O,l)-forms is the inverse of the complex Laplacian dd* + d*d. N and its regularity theory play 
a crucial role both in partial differential equations and in several complex variables (^0], [2], [HI)- 
In particular, the question when is a compact operator on C'^^^-^ii^), the space of (O,l)-forms 
with square integrable coefficients, is of interest in several contexts. Well known examples are 
global regularity and the Fredholm theory of Toeplitz operators. We refer the reader to ^2] for 
more information and for references on these and other questions related to compactness of the d- 
Neumann operator. Among more recent references, we mention ^1], where compactness is related 
to the existence of Henkin-Ramirez type integral formulas with well behaved kernels. And, there is 
a useful connection between compactness in the 9-Neumann problem and the asymptotic behavior, 
in a semi-classical limit, of the ground state energy of certain magnetic Schrodinger operators 
0). 

The most general known sufficient condition for compactness is potential theoretic in nature. 
Roughly speaking, there should exist, near the boundary points of infinite type, plurisubharmonic 
functions with arbitrarily large complex Hessians whose gradients are uniformly bounded in the 
metric induced by the Hessians of the functions. This condition was introduced under the name 
property (P) and shown to imply compactness by McNeal in ^H]- It generalizes previous work by 
Catlin that used his now classical condition property (P) (0], see for a systematic study of this 
property). Property (P) implies compactness of the 5-Neumann operator on an arbitrary bounded 
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pseudoconvex domain, even when no boundary regularity whatsoever is assumed (^Hli Corollary 
3; [IS], Corollary 4.2). 

There are natural geometric conditions, bearing on how the set of infinite type boundary points 
sits inside the boundary, which are known to imply property (P) or (P) and hence compactness 
of the 5- Neumann operator (^, ^7], P, ^2]) EH])- In this paper, we present new geometric 
conditions which imply compactness in the case of smooth bounded pseudoconvex domains in C^. 
In light of the discussion in the next paragraph, the fact that compactness is not demonstrated via 
establishing property (P) or (P) (in contrast to ^7j> CP' C2i C2> CHI) is worth pointing out. 

On domains which are locally convexifiable, the analysis, the potential theory, and the geometry 
associated with the 9-Neumann problem mesh perfectly. That is, the following three statements 
are equivalent: (i) the 9-Neumann operator is compact, (ii) the boundary of the domain satisfies 
property (P), (iii) the boundary of the domain does not contain (germs of) analytic discs (jllj. 
|12j . Theorem 5.1). This is far from true in general. Property(P) always excludes discs from the 
boundary (as is seen by pulling back the good plurisubharmonic functions to the unit disc in the 
complex plane); so does compactness of N for domains in C'^ with at least Lipschitz boundary (see 
|12j . Proposition 4.1 for a proof of this folklore result). However, Sibony (^Tj) observed that the 
absence of discs from the boundary need not imply property (P) , and Matheos ( ; see also ^2] ) 
showed that in fact the absence of discs need not (even) imply compactness of the 5-Neumann 
operator. (Actually, Sibony's observation concerned property (P), but the domains in are Har- 
togs domains in where (P) and (P) are equivalent {^^, Appendix A).) This left open the exact 
relationship between property (P) and compactness of the 9-Neumann operator. Christ and Fu 
([7j) recently established the equivalence of these two properties on smooth bounded pseudoconvex 
Hartogs domains in C^. On general domains, however, the situation is not understood. In view of 
this, it is desirable to have a technique like ours for establishing compactness of the (9-Neumann 
operator that does not rely on property (P) (in this context, compare also |14j). 

If Z is a (real) vector field defined on some open subset of bO, (or of C^), we denote by J-^ 
the flow generated by Z. By finite type of a boundary point, we mean finite type in the sense 
of D'Angelo; however, for domains in C^, the various notions of finite type all agree, so that no 
distinction is necessary (|H1)- Recall that the set of points of finite type in the boundary of a 
smooth bounded pseudoconvex domain is open and, consequently, the set of points of infinite type 
is compact. P(P, r) denotes the open ball of radius r centered at P. 

Theorem. Let Q be a -smooth bounded pseudoconvex domain in C^. Denote by K the set of 
boundary points of infinite type. Assume that there exist constants Ci,C2 > 0, C3 with 1 < C3 < 
3/2, and a sequence {sj > 0}?^i with lim £j = so that the following holds. For every j £ N 
and P £ K there is a (real) complex tangential vector field Zpj of unit length defined in some 
neighborhood of P in 6$! with max \div Zpj\ < Ci such that J^J^ . (P(P, C2(ej)'^3) n K) Q bQ\K. 
Then the d-Neumann operator on is compact. 

It is of course assumed that the flow J^Zp Grists for all initial points in P(P, C2(ej)*~'^) f\K and 

t < 8,. 
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Two immediate questions arise. First, the obvious examples that satisfy the assumptions in the 
theorem also satisfy property (P), and whether or not the theorem can actually furnish examples 
(if they exist) of domains with compact (9-Neumann operator, but without property (P), does not 
appear to be a simple matter to decide. But we obtain, in any event, a simple geometric proof 
of compactness under the conditions in the theorem. Moreover, these conditions turn out to be 
natural and, modulo the form of the lower bound C2(ej)'^^, minimal; we discuss this in Remark 3 
below. Second, our result is stated for domains in C^; the proof uses maximal estimates and so does 
not work in higher dimensions. However, it is easy to see that the theorem, when transcribed to C" 
verbatim, also fails. Control over K will have to be imposed in all complex tangential directions 
rather than just in one. We hope to return to these questions elsewhere. 

The general thrust in the theorem is that at points of K there should exist a (real) complex 
tangential direction transversal to K in which bQ\K (the good set) is thick enough. This occurs in 
extremis when K is (locally) contained in a totally real submanifold of the boundary (property(P), 
and hence compactness, are well known in this case (lU)). The following corollary to the theorem 
takes this situation and disposes of the requirement that K be (contained in) a smooth submanifold 
of the boundary. 

Corollary 1. Let fl be a C°° -smooth bounded pseudoconvex domain in C^. Denote by K the set of 
boundary points of infinite type. Assume that for all P ^ K there exists a (real) complex tangential 
vector field Zp defined near P, a neighborhood Up of P, and > such that {Up fl K) CI bO,\K 
for t < Ep. Then the d-Neumann operator on 0, is compact. 

To reduce Corollary 1 to the theorem, it suffices to cover K by finitely many neighborhoods 

Up. 

A second geometrically simple special case occurs when bO,\K satisfies what might be called a 
weak complex tangential cone condition. That is, there should exist a finite (possibly small) open 
real cone C in ~ having the following property. For each P a K there exist a complex 
tangential direction so that when C is moved by a rigid motion to have vertex at P and axis in that 
complex tangential direction, the (open) cone obtained intersects 617 in a set contained in b^l\K. 
Then the assumptions in the theorem are satisfied with C3 = 1. 

Corollary 2. Let be a C°° -smooth bounded pseudoconvex domain in C^. Denote by K the set 
of boundary points of infinite type. Assume that bn\K satisfies a weak complex tangential cone 
condition. Then the d-Neumann operator on Q is compact. 

Because C3 is only required to be in the range 1 < C3 < 3/2, the cone in Corollary 2 may be 
allowed to degenerate into a mild cusp; we leave the details to the reader. 

Remark 1. The assumption in the theorem says that for all j € N, all P K, there should exist a 
complex tangential vector field and a ball centered at P so that the intersection of K with this ball, 
when translated along the integral curves of the vector field by £j, ends up in bQ\K, i.e. in the set 
of points of finite type. The crucial additional condition is that the radius of this ball should be at 
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least of order {sj)^ for some C3 < 3/2. A condition of this kind is needed; it is not enough to just 
have some ball centered at P. Consider a Hartogs domain Q := {{z, w) G C^/|z[ < 1, \w\ < e"'^^^''}, 
where (p is smooth and plurisubharmonic on the unit disc, and such that is a smooth domain in 
C^, which is strictly pseudoconvex at boundary points where w = 0. Let Kq := {z ^ D \ Acp = 0} 
be relatively compact in D. Assume that A(p vanishes to infinite order on Kq. It is well known that 
the set K of boundary points of infinite type consists precisely of those {z,w) G bO, with z G Kq. 
Finally, assume that Kq has empty interior (this is equivalent to 60 not containing analytic discs). 
li P = {z,w) G K CI bfl, then z G Kq. Because Kq has empty interior, there are arbitrarily short 
translates of z in D\Kq. Because D\Kq is open, there exists for every e > a vector X and a ball 
B(z,r) such that B{z + X,r) C D\Kq. The (constant) vector field X is easily lifted to a complex 
tangential vector field on bO, (near P). In this way, one sees that satisfies the assumptions of 
the theorem, except for the lower bound (in terms of £j) of the balls centered at points of K. This 
is enough to make the theorem fail: there are examples of Hartogs domains as above, where the 
9-Neumann operator is not compact (^Sj) d]i Theorem 4.2 and Remark 5). 

Remark 2. The key to getting the vector fields from the theorem in the examples in Remark 1 is 
that Kq has empty interior, or equivalently, that bO, contains no analytic discs. It turns out that 
suitable vector fields may be obtained on any domain in whose boundary contains no analytic 
disc. In fact, the boundary of a smooth bounded pseudoconvex domain in contains no analytic 
discs if and only if it satisfies all the assumptions of the theorem, except possibly the lower bounds 
(in terms of £j) on the radii of the balls centered at points P £ K. It is clear that the assumptions in 
the theorem, without these lower bounds, exclude discs from the boundary. The proof of the other 
direction follows easily from Proposition 3.1.12 in ([2]); as follows. Let P G bO,, Z a real vector field 
defined on bO, in a neighborhood of P that is complex tangential, and let J denote the involution 
associated with the complex structure of C". Then Z and J{Z) span (over R) the complex tangent 
space to bQ near P. For < 6* < 27r, denote by the field := (cos 6*)^ + (sin(^) J(Z). Fix e > 0. 
Catlin shows in Proposition 3.1.12 that if ah points of := {T^^eiP) \ < t < e,0 < 9 < 2Tr} 
are weakly pseudoconvex points of bO,, then is a (necessarily one-complex dimensional) complex 
manifold. Consequently, when there is no analytic disc in bi}, there exist complex tangential fields 
Zp^^ for all P € K and sufficiently small e > so that J'Zp {^) ^ ^ ^ close enough to P. 
Moreover, the Zp^^ are of the form Zp^^ = {cos9p^e)Z + (sin ^p^^) J(Z), so that div Zp^^ is bounded 
uniformly in P and e. 

Remark 3. Since on a smooth bounded pseudoconvex domain in C^, compactness of the 9-Neumann 
operator implies that there are no discs in the boundary (see the discussion preceeding the statement 
of the theorem), we can deduce from Remark 2 a (very) partial converse to the theorem. If N is 
compact, then the domain satisfies the assumptions in the theorem except possibly the lower bound 
C2(ej)'^^ on the radii of the balls whose intersections with K, when translated along the integral 
curves of the fields Zpj-, end up in bQ\K. Moreover, the examples in Remark 1 show that there 
must be some lower bound on the size of these balls in terms of £j, otherwise the theorem need not 
hold. Thus the assumptions in the theorem are quite natural, and, modulo the exact form of this 
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lower bound, minimal, as asserted in the discussion following the statement of the theorem. 

We now prove the theorem. There are various equivalent statements to compactness of the 
9-Neumann operator, see e.g. Lemma 1.1. We will show a so-called compactness estimate, 
that is, we will show that for all e > there exists a constant Cg such that 

\\u\\l<e{\\du\\l + \\d*u\\l) + Ce\\uf_^, uGdom(9*)nC(^^i)(n). (1) 

Here, C^^-j{Q) denotes the space of (O,l)-forms with coefficients in C°^{Q), the norm || ||o and 
II ||_i denote the Sobolev-0 (i.e. £?{^l)) and Sobolev-1 norms, respectively. For forms, these norms 
are computed componentwise. Note that if (0) holds for u G dom(5*) n C^^,^{Q), it holds for 
u G dom(9*) n dom((9), because the former space is dense in the latter with respect to the graph 
norm (see for example jH], Lemma 4.3.2). 

The idea of the proof is very simple. To estimate the £^-norm of u near a point P of K, we 
express u there in terms of u in a patch which meets the boundary in a relatively compact subset 
of the set of finite type points plus the integral of the derivative of u in the direction Zpj. The first 
contribution is easily handled by subelliptic estimates, while the second is estimated by the length 
of the curve (which is ej) times the >C^-norm of Zpju. But in C'^, this >C^-norm is estimated by the 
£^-norm of du and d*u, because Zpj is complex tangential. Controlling the terms coming from 
the integral of Zpju raises overlap and divergence issues; these are taken care of by the uniformity 
built into the assumption in the theorem. 

The details are as follows. First note that we can extend the fields Zpj form bfi to the inside 
of by a fixed distance by letting them be constant along the real normal. We still denote these 
extended fields by Zpj; they are complex tangential to the level sets of the boundary distance. 

Fix e > and j so that ej < e. A standard covering theorem (see e.g. Theorem 1.3.1) 
apphed to the family of closed balls {B{P, ^{ej)'^^) \ P e K} gives a subfamily of pairwise 
disjoint balls so that the corresponding closed balls of radius ^{ej)'^^, hence the open balls of 
radius C2(ej)*^^, still cover K. Because K is compact, we obtain a finite family of open balls 
{5(Pfc,C2(ej)^^) \ I < k < N,Pk e K} that covers K, and such that the corresponding closed 
balls of radius j^i^j)'^'^ are pairwise disjoint. To simplify notation, we will use Z^ to denote the fields 
Zp^j, 1 < k < N . By decreasing C2 in the theorem, we may assume that J^^^{B{Pk, C2{£j)'^^)^K) 
is not only contained in bQ\K, but is relatively compact there. Consequently there exist open 
subsets Uk, 1 < k < N , of Q, with 

K n B{Pk, C2{ejf') CUk^ B{Pk, C2{ejfn (2) 

and 

j^z',{Ukr\n)r\K = (li. (3) 

It is implicit in the statement of (jHI that Uk is chosen in a sufficiently small neighborhood of bQ so 
that the flow generated by the extended vector fields Z^ exists up to time ej for all initial points 
in Uk- 
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Now let u G C(^_i)(J^) n dom{d*). Then 



kilo 



\U\' 



(4) 



N 



Because \^ ^ |J Ukj meets bQ in a compact subset of b^l which does not intersect K, we can 
apply subelliptic estimates (see j^]) to estimate the second term in the right-hand side of Q): there 
exist s > and C > such that the restriction of u to a neighborhood U of 17 \^ ^ |J U/^ (in 0) 
belongs to W^q (U) and 

||u||^.^^^^(C/) <C(||an||2 + ||a*u||2). (5) 
The usual interpolation inequality for Sobolev norms gives 



< ll^ll£^o,,)(C/) 



< ^llti 



+ CJ\u\ 



(6) 



<e{\\du\\l + \\d*u\\l) + Ce\\uf_^. 
We now estimate the first term on the right-hand side of Q. Fix k, 1 < k < N . 



Uknn u^nn 







dV{x) 



<2 j \u{:^l{x))\''dV{x) 



+ 2 



Uknn 



Zku{T'zS^))dt 



dV{x). 



The first term on the right-hand side of ^ can be estimated as follows: 



\u{J^J{x))\^dV{x) 



Uknn 



rMUknn) 



<Ck 



\u{y)\^ <lei{dx/dy)dV{y) 
\u{y)\''dV{y). 



rMUknn) 



(7) 



(8) 
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Here we use det{dx /dy) as shorthand for the (positive) Jacobian of the diffeomorphism ^z^^ ■ 

we can use subehiptic estimates once more to estimate the last 
term in Q; an argument analogous to the one that led to © gives 

\u{y)\'dV{y) < ^i\\Ou\\l + \\d*u\\l) + (9) 

We now estimate the second term on the right-hand side of (0) . The Cauchy-Schwarz inequality 
and Fubini's theorem give 



dV{x)<ej / \Zku{TUx))\^dt dV{x) 



Ukfin 



Uknn 



\ZkuiTiix))\^dVix) dt 



\Zku{y)\^detidx/dy)dViy) dt 



n, {UkDU) 



< 2ei 



\Zku{y)\^dV{y) dt. 



j^i^iUkfin) 



(10) 



In the last inequality in (jlUj) we have used the uniform bound on the divergence of the fields Z^, 
that is, on the rate of change of the volume element under the flows generated by the Z^'s. This 
bound implies that det{dx / dy) < exp(fCi) < exp(ejCi) < exp(eCi) < 2 for e small enough. (It 
suffices to establish (0) for small enough e > 0.) 

Putting together estimates (f7|)- (|lU() and adding over k, we estimate the first term on the right- 
hand side of(jlJ): 



X A:=lrr 



JV 

U Uk]nn 

fe=i 



N 
k=l 



^i\\du\\i+\\d*u\\i)+Cs\\ue, 



(11) 



+ AejJ J \Zku{y)\'dV{y) dt 
:F'z(Uknn) 
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That is, 



/ 



<2ei\\du\\l + \\d*u\\l) + Cs 



we. 



N 

U Uk]nn 



N 

E 

k=l 



(12) 



\Zkuiy)\^dViy) 



Note that we adopt the usual convention of denoting by constants which depend only on e, but 
whose actual value may change as the estimates progress. 

No point of Q is contained in more than C(C2)(ej)^~^'^^ of the sets T\^{Uk 1 < k < N, 

where C{C2) denotes a constant depending only on C2. Indeed, if Q € J^^(f//fc nil) n^^(f/m nil), 
then by the triangle inequality, the distance between T'^^{Q) and J-'^^{Q) is no more than 2t < 2ej 
(recall that \Zk\ = 1). Consequently, B{Pm, ^ B{Pk,2ej + 2C2{ej)'^^ + %{ej)'^'^). Since 

the balls B{Pm, j^iejf--'), I < m < N, are pairwise disjoint, comparison of volumes gives the 
desired upper bound on how many of them can be contained in B{Pk,2ej + 2C2{ej)'^^ + ^(ej)'^^). 

With this control over the overlap of the sets T^^{Uk riQ), 1 < k < N, for t fixed, we can 
control the last term in (|12j) . Denote by L the complex tangential field of type (1,0); we may take 
L to be C°° on Jl, and so that |Re L\ and |Im L\ are one in a neighborhood of the boundary that 
contains all the sets UkCiQ, 1 < k < N. Then, because \Zi^\ = 1, 

\Zku{y)f < 2|(Re L)u{y)\^ + 2|(Im L)u{y)\^ . (13) 

Inserting ((T^ into ((T^ and using the estimate on the overlap of the sets (Uk n Jl), we obtain 



\u\'' < 2e{\\du\\l + \\d*u\\l) + Ce\\u\\'i^ 



' N 

U Uk I nn 

. fc=i 



+ As, 1 2C(C2)(e,)^-4^^ I (|(Re L)u{y)\' + |(Im L)u{y)\^)dV{y) 



dt 



(14) 



2e{\\du\\^Q + \\d*uU) + CeMii 



8CXC2)e^-^^^ I (|(Re L)u{y)\^ + |(Im L)uiy)\^)dV{y). 



Finally, we exploit maximal estimates {^)- since we are in C^, the integral in the last term in 1)14(1 
is dominated by ||5ii|lQ + ||9*ti||Q. Combining this with @ and © shows that there is a constant 
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C independent of e such that for all sufficiently small e > 0, we have the estimate 

\\u\\l < Cie + e^-''^'){\\du\\l + \\d*u\\l) + Ce\\uf__^. (15) 

((T3|) gives the required compactness estimate (0) (since C does not depend on e and 6 — 4C3 > 0). 
This completes the proof of the theorem. 
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